The project was distinguished as one of 6 excellent solutions among 180 participants in Tsinghua online competition. (Within top 3.3%)

Advisor: Prof. Xiaowo Wang.

task definition

solution

a simple idea:

many successful solutions used CLIP model, which is a popular visual-language pretraining model.

CLIP is designed for understanding phrases or sentences, but here we notice that the text prompt is as simple as single Chinese words. thus sequential text encoder could be replaced by simple word embedding-mask dictionary for efficiency.

pipeline illustration:

performance:

single-image-level accuracy ~92%

group-level accuracy ~84%

code and report (in Chinese)

code: here

report (in Chinese):